Evaluating Reference Services in the Electronic Age

INA N ELECTRONIC ERA, THE EVALUATION OF REFERENCE and related information services should still be based on the same principles used to evaluate traditional face-to-face reference services and printed reference tools. Traditional research methods-which are surveys and questionnaires, observation, individual and focus group interviews, and case studies-can be utilized very effectively in an electronic environment. However, electronic technologies offer interesting research opportunities not present in the traditional reference environment. INTRODUCTION At conferences and workshops on evaluating reference services, the most frequent recurring question librarians ask is, “How can the material on evaluating reference services be applied to assessing electronic reference services?” The best answer is, “Take existing methods, determine which will best meet the study goals, and then adapt those methods to the electronic environment.” In any environment, evaluating reference services still requires starting by assessing why reference services are being evaluated and what the organization plans to do with the study results. Before trying to decide how to evaluate electronic services, performance standards that set the level of achievement expected for the service should be explicitly stated. In determining the performance standards to be adopted, the organization must decide what values are crucial. Are members of the organization concerned primarily with Jo Bell Whitlatch, Reference Librarian & History Selector, San Jose State University, SanJose CA95192-0028 LIBRARYTRENDS, Vol. 50, No. 2, Fall 2001, pp. 207-217 02001 The Board of Trustees, University of Illinois 208 LIBRARY TRENDS/FALL 2001 I. Economics-the cost or productivity of services; 2. The process-aspects of librarian/reference system and user interaction; 3. Resources-books, indexes, databases, staffing levels, equipment, design of physical or electronic environment; or 4. Products/outcomes-information or knowledge that the users obtain. In an electronic environment the interactions between librarians and users often will no longer be truly face to face. Thus, process standards are the measures that most need to be reviewed in a digital reference em‘ 71ronment. Librarian behaviors that are crucial in the reference-desk environment will need to be redefined for remote reference services. Work on redefining process standards has already begun. The Virtual Reference Desk (VRD) project has developed a list of User Transaction Standards to address aspects of librarian/system and user interaction. The standards address several “facets” related to quality: accessible, prompt turnaround, clear response policy, interactive and instructive (Kasowitz, Bennett & Lankes, 2000). Most of these facets address the process standards, rather than standards related to economics, resources, or products/outcornes. In a remote electronic reference enhironment, accessibility and prompt turnaround could become dominant in user evaluations. Miwa (2000) used digital reference serllces features of acknoivledgnien t, responsiveness, and tone of message to represent the process aspects of the reference interaction in a digital environment. She also looks at user situations as part of the process-for example, wording of the request by the user and user’s ability to comprehend the message. Broad goals for the study should be prepared in writing once a reasonable degree of consensus has been achieved on the particular set of standards that an organization wishes to emphasize. After broad goals have been developed, written objectives should be developed for each study goal. The objectives should be measurable so that, at the conclusion of the evaluation, one can identify any gaps between the present level and the desired level of reference service performance. This present paper discusses how to apply traditional evaluation methods in an electronic reference environment once the study goals and objectives have been determined. Readers desiring additional information on setting performance standards and developing goals and objectives for reference service evaluation may wish to consult Evaluating Refkrence S m i r es: A Practical Guide (Whitlatch, 2000). All methods have strengths and weaknesses. Depending on the goals and objectives of the study, some methods will be more effective than others. As a general rule, utilizing more than one method is recommended in a single study, because the strengths of one method often compensate for the weaknesses of another. The advantages and disadvantages of the various methods may also change somewhat in an electronic environment. This WHITLATCH/EVALUATING REFERENCE SERVICES 209 paper considers how applying surveys, observation, interviews, and case studies-all traditional evaluation methods used in assessing face-to-face services-presents new opportunities and challenges in assessing electronic reference services. SURVEYS AND QUESTIONNAIRES Surveys or questionnaires are methods of directly collecting information on individuals’ thoughts, beliefs, attitudes, and opinions, plus objective data, such as education, gender, and income. The survey method has been the most frequent way of assessing traditional reference services. In the past, surveys have been relied upon too heavily because they are the most efficient method of assessing a large group of representative users. Also. for the inexperienced researcher, surveys appear easy to design. The disadvantages, such as obtaining meaningless information from poorly designed questions and the lack of depth of information from standardized responses, are often not appreciated until too late. Another significant problem in using surveys is low response rates, particularly from surveys distributed through the mail. A substantial number of nonrespondents can bias the results; those who choose not to complete the survey might hold very different views from those who do. Internet questionnaires can be used effectively to survey attitudes and opinions on the quality of reference service related to process (the interaction with the virtual reference service) and products/outcomes (the value of the information obtained). An Internet survey asking for an evaluation of service provided can be sent out within days after the user has received an answer. In contrast to surveys distributed in person at the reference desk or in the library, emailing the questionnaire can also be calculated to allow most users some time to use and further evaluate the information obtained through a reference interaction. As Zhang (1999) points out, the Web provides new opportunities to conduct survey research more efficiently. Research costs for sending out Internet surveys are relatively low and the turn-around time short compared to conventional mail-in surveys. Also, email can be used effectively to follow up on paper-based surveys (Roselle 8c Neufeld, 1998). Most responses received in electronic format have been precoded, eliminating transcription errors and saving time and expense. McCullough (1998) notes that Web-based surveys are faster, generate more accurate information, and cost less. He has found that a respondent will typically complete a Web-based survey in about half the time it would take an interviewer to conduct that survey by telephone or in person. Resolving the technical problems with Internet surveys requires greater technical expertise on the part of the researcher than does research conducted with traditional survey methods. However, services that provide Web survey forms and guidance to assist researchers in designing and de210 LIBRARY TRENDS/FALL 2001 veloping Internet surveys are becoming common. Names and Web addresses for some of the services that have been positively discussed on the Academy of Management Research Listsen: rmnet@list<sspru.nc.edu, are provided in the Appendix. Zhang (1999) also reviews potential problems and concerns related to Internet-based surveys. One of the greatest strengths of survey research is the ability to randomly select respondents in a manner that ensures a sample representative of the target population. In telephone surveys, respondents are randomly selected, but most online poll respondents are selfselected (Pew Research Center, 1999). The greatest difficulties with Internet surveys occur when the survey does riot reach certain types of respondents who need to be included in the survey population. Biased samples and returns can be a major problem because certain social groups are underrepresented among Internet users. However; for surveying users of electronic reference services, bias should be minimal. Respondents must have access to the Internet in order to use the electronic senices; they can presumably access a Web survey form as well. Some individuals may not have convenient access from their home or office and may use the service only occasionally from an Internet cafe or a library. If these individuals are not identified, this group may be underrepresented. Individuals who do not have convenient access may, as a whole, be less experienced users of electronic reference services. If these users are not included in the sample, survey results may not truly represent the population as a whole. Other means, such as a telephone interview or mail survey, may be required to obtain responses from them. Finally, if the purpose of the survey is to collect information from people who do not use electronic sources, reljing upon the Internet as the principal method of survey delivery will present a very serious problem. In addition, low response rates are a serious problem with Internet surveys. In her evaluation of AskERIC, Shostack (2000) observed that users were either extremely happy or dissatisfied with digital reference services. These results suggest that only motivated users are responding. A study that replicated an earlier study found a disturbing decline in email response rates: in 1995 the email response rate was 80 percent, but by 1998 it had fallen to 42 percent (Bachmann, Elfrink & Vazzana, 1999). The researchers suggest that the most likely reason for the decline is the respondents’ increased reluctance to respond by email. Zhang (1999) concludes that the Internet cannot serve as the only means to collect survey data if researchers need representative returns from a sample. Schaefer and Dillman (1998) found that giving advance notice requesting participation generally increases response rates. The Pew Research Center (1999) has tested an interesting approach. Email addresses were collected from individuals who were called aspart of randomly selected national samples. If these individuals agreed to participate in a future onWHITLATCH/EVALUATING REFERENCE SERVICES 211 line survey, their email addresses were placed in a pool. Then, in a second phase, a random sample was selected from this pool. Email addresses were used for verification purposes to prevent respondents from taking the survey more than once. McCullough (1998) suggests that the questionnaire be posted on a secure Web site. Respondents can be generated from personal invitations issued by email. He notes that a sufficiently large sample of 300 or 400 respondents can often be completed over a weekend. In order to apply scientifically tested polling techniques to Internet technologies, Stanford political scientists Douglas Rivers and Norman Nie have created Knowledge Networks. With $42 million in venture capital, they have installed free WebTV devices normally costing $250 each in 40,000 homes selected through random phone calls. Because everyone in the household nineteen or older is involved, there are about 100,000 participants. The homes receive a black box slightly smaller than a VCR, a cordless keyboard, and many instructions. The homes are expected to remain in the survey pool for three years. In exchange for answering brief surveys about once a week, the households receive free Internet access, email, and frequent chances to win prizes. Of those who were asked tojoin the Knowledge Networks pool, 56 percent agreed-compared with 15 percent of people who usually agree to participate in phone polls. Although the polling is a significant activity, the primary company income comes from consumer research for manufacturers (Konigsmark, 2000). Zhang (1999) also reports that validity of Internet survey responses can be adversely affected. Unintended participants may respond because of the ease of forwarding email messages to other people. Individuals can respond to a single survey by submitting the same reply many times. Unique caseidentification numbers should be assigned to each respondent to control for multiple responses and unintended participants. Nondeliverable surveys are also a major disadvantage of email. In 1995 and 1998 studies, Bachmann, Elfrink and Vazzana (1999) found that about 20 percent of all emailed surveys were nondeliverable. Comfort level with the Internet survey form should also be considered. Zhang (1999) found that, while 80 percent of usable replies were received via the Web, 20 percent of respondents chose to complete the survey via postal mail or fax. Internet survey respondents did report problems with the layout of the survey questionnaire on low-resolution monitors, problems going back to previous parts of the questionnaire, problems with printing, and (on computers with low-speed modems) problems with downloading the questionnaire. Users also reported that comments were also more difficult to insert on electronic survey forms than on paper forms. Shostack (2000) also noted a tendency for users to ignore open-ended questions on Internet survey forms. (This problem is not unique to online surveys. In the author’s experience, most users completing paper forms also tend to leave openended questions blank.) Surveys not conducted by telephone or in-person 212 LIBRARY TRENDS/FALL 2 0 0 1 interview tend to have rather limited potential to collect qualitative data. An experiment with incentives in the form of cash prizes revealed that, while the overall iiumbers of respondents did not increase significantly, the number of conipleted Internet survey questionnaires did rise (Pitkow &Kehoe, 1996).